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© Multiple frame motion estimation. 



© The invention relates to developing a signal to represent each block of an image contained in a frame being 
interpolated in a motion interpolated system such that the error between the frame being interpolated and the 
reference frames is minimized without placing any predetermined limitation on the values of either the weights or 
the displacements employed as motion vectors. The weights and displacements required to achieve the 
minimization of the interpolation error are jointly determined on a block by block basis. A set of groups having a 
candidate displacement from each reference frame and, in accordance with an aspect of the invention, their 
corresponding best weights are determined. An error signal which would result if each member of the set of 
groups of candidate displacements from each reference frame and their corresponding best weights were 
employed to represent the block is evaluated. The group having the lowest error signal is either selected to 
represent the block or as a starting point for the selection of a new set of groups. By employing other aspects of 

^ the invention, the number of groups which need be evaluated to represent the block can be reduced from a 

^ predetermined maximum number. 
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This invention relates to video image processing, and more particularly, to a method for est! mating the 
motion of image blocks when more than one reference frame is employed in a mot,on P red,ct,ve -mage 
processing system. 

Background of the Invention 

7Z^ ££SZL ». ***** * . -™ — * - p™—* «"° ded re,eren ~ ,ra ™ 

^SSrd2S£L-« » bo .ployed « *. motion vector , soar* .or fe. W- one, - • - o< 

the minimization of the integral of the absolute error s.gnal, i.e.. 

min £ |l„0O-In-l(*' + d)| (D 

d »'€N(X) 

^'^SSZSSToTI^ghB utilized to, the waighfed sum is typically performed by employing 
oitb™,^ 

TTJZ£Z?££2?~ Te P^Toon*^ trom l« on. o, the reference frame. ,0.,, 
andTo? (Sfllo MPE6 d?»« propose,). The motion vectors, .no, sign., and weighting factors fo, ..oh 

nrS"S^rS^?i5^ ~ advantage o, fee av.l^e oand„,dfe and 
therefore require a higher bandwidth to provide an optimal reconstructed .mage. 
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Summary of the Invention 

The invention as defined in claim 1 . 
Brief Description of the Drawing 
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In the Drawing: 

A simplified block diagram of an exemplary transform video encoder embodying aspects of the 
invention is shown in FIG. 1; 

FIG. 2 illustrates aspects of a block-matching search process between two frames; 
5 Shown in FIG. 3 is a weighted averaging process that occurs in an encoding system employing 2 

reference frames; 

FIG. 4 shows aspects of 3-step search strategy that can be employed when using 2 reference frames; 

Shown in FIG. 5 is a portion of frame l n divided into blocks of pels; 

Illustrated in FIG. 6 is a threshold search strategy with ordered scanning; 
to FIGs. 7, 8, and 9 .when connected together, form a flow diagram of a conjugate direction search 

strategy that has been extended, in accordance with an aspect of the invention, for use with 2 reference 
frames; and 

Shown in FIG. 10, in flow chart form, is a simplified composite, in accordance with an aspect of the 
invention, of the steps required by a motion estimator to generate motion vectors and weights for a block to 
t'j be encoded. 

Detailed Description 

FIG. 1 shows, in simplified block diagram form, exemplary transform video coder 100 for encoding a 

po sequence of image frames (frames) represented in digital form, signal VIDIN, in accordance with aspects of 
the invention. One aspect of the simplification, for clarity of exposition purposes, is that storage elements 
which may be embedded in each element shown that are required for computation or sequencing and 
coordination purposes are not shown. However, use of such storage elements will be readily apparent to 
one skilled in the art. Also, an overall timing and sequencing controller is not shown. Again, the use of such 

25 a timing and sequencing controller will be readily apparent to one skilled in the art. 

Transform video coder 100 supplies as an output an encoded digital data signal, BITSTREAM. that is 
representative of the frames of signal VIDIN with which it was supplied as an input. Each frame so supplied 
can be encoded by transform video coder 100 via one of three (3) coding methods: intraframe coding, 
motion predicted coding and motion predicted interpolated coding. Signal VIDIN is supplied as an input to 

30 control and frame rearrangement unit 101. Control and frame rearrangement unit 101 controls the overall 
operations of video coder 100 by determining via which method a frame should be encoded based on the 
temporal position of the frame. For purposes of this example, the type of encoding for any particular frame 
is indicated as follows: a zero (0) indicates to encode the frame with intraframe coding, a one (1) indicates 
to encode the frame with motion compensated prediction coding and a two (2) indicates to encode the 

35 frame with multiple frame motion interpolated coding in accordance with an aspect of the invention. 
Intraframe and motion compensated prediction coding are well know. 

Control and frame rearrangement unit 101 temporarily stores a predetermined number of frames of 
signal VIDIN so that the frames can be supplied as output in a different temporal order than that in which 
they were received. Each frame, the structure of which is a lattice of pe\s each of which has a particular 

40 location, is divided into QxR blocks of pels. These pel blocks are supplied in a predetermined order as 
signal BLOCKS. Such a resequencing is employed so that the at least two reference frames which are 
required for motion estimation and interpolation can be processed prior to the processing of any frames 
which are to be motion interpolated encoded. The encoding type determined by control and frame 
rearrangement unit 101 is conveyed over first control bus 102 to switch elements 103 through 110. Block 

45 type unit 111 decides on a block by block basis whether a block of pels should be intraframe encoded 
despite the fact that the frame to which it belongs is designated to be either motion compensated predicted 
or motion interpolated encoded based on the temporal position of the frame in video signal VIDIN by control 
and frame rearrangement unit 101. Such a decision is based on block type unit 111 determining that 
intraframe encoding would require less bits to represent the block than the other types of encoding. To this 

so end block type unit 111 is supplied with signal BLOCKS and the error of a reconstructed motion 
compensated predicted or motion interpolated version of the block with respect to the actual block, signal E, 
supplied by subtracter 112. Block type unit 111 controls switch elements 113 and 114 by sending 
commands over second control bus 115. In addition, variable length coder 116 is responsive to commands 
from block type unit 1 1 1 to suppress the supplying of motion vectors or weights to multiplex and buffer unit 

55 117. 

Any element of transform video coder 100 which does not receive during the processing of a block any 
input that is required to generate its associated output is considered to have no operation during the 
processing of that block. Such inputs may not be received because of the position of one or more of switch 

3 
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elements 102 through 109 and wrtc ^ f ' e 7"? 1 ' ^ that sign a. VID.N is first being received after 
For clarity of expos.fon purposes ,t is assumed that g ^ ^ & ^ ^ frame 

initialization of transform v.deo coder 100 o Jha rans orm ^ m 

encoded via intraframe coding at |>aMm^ nterva^s ^and s pres „, 114 has 

intraframe coding mode each of swrtc ^ leme "^ 3 J h ^ s , a zero (0) in its associated label or if the 
their arm positioned to connect to the terminal ^J^^^L ^on of the switch is irrelevant, 
switch has no terminal with at least a ^^^ZZVT^ through switch element 103 and 114 
Signal BLOCKS passes from centre and ^ t ^S^L of pels are" then converted from the pe. 
to discrete cosine transform un.t (DCT) 119. The uxn produces and supplies as an 

domain to the Discrete Cosine Transform domain by DCT I ^DCT p ^ ^ ^ ^ 

output a set of frequency coefficients that are grouped ^ 2 J^™ * DCT 119 are then supplied to 
b.ocks received as input. The transform ^^^^^iluXs to assume. The quantizing 
quantizer 120. A quantizer ^^J^^^^^^S stored in multiplex and buffer unit 
can optionally be made response to the quantity of ,nto ^ ^ ^ ^ ^ supphed 

signa. QNTCTRL. Quantized coefficients supp ^ ™ ™™XanaUe coding is well known and the 

an input to variable length coder (VLC) 121 J^JJJ « representation of the 

choice of coding methodology is at the ^* s u nit 117. Multiplex and buffer unit 117 

quantized coefficients is supplied as an «J^ U J^ "^coeflw^ in multiplexed fashion with 
subsequently supplies the encoded "P""*"" * BfTSTREAM which may be transmitted or 

the outputs of variable length coders 116 and 122 as s.gna. c 

stored, depending on the application Quan tizer 120 are also supplied as an input to 

The quantized coefficients W«« "J "SSJSr 123 performs an inverse quantization on the 
dequantizer 123 through switch element ° 5 nn ^"~ efficie( J s wnich are supplied to inverse discrete 
quantized coefficients thereby generating ^^^^St^ the block in the pel domain. This 
transform unit (.DCT) 124 to ge nerate 125 through adder 126 and switch 106. 
reconstructed block is suppl.ed to and stored m frame .butter £ > ^ ^ buffer 

Adder 126 is supplied with a nul. (a» zero) boc ^^^^^ or interp0 ,ated. 
125 becomes a reference frame m the past U oMra mesto b p ement unit 101 supplie s as 

Upon conclusion of process.ng °\^ e ^^ t ?^Xo) in the futur e relative to the first frame and all 
output signal BLOCKS QxR blocks of a second frame (lp *.n the . u > md& the contro , of 

frames tobe motion interpolated until a new base J-jjnod. switc / elernents 103 through 

control and frame rearrangement un.t 101, over first contro dus . associated label or if 

110 has their arm positioned to connect to ^ ■^ , JEJ££5 the switch is irrelevant. For clarity of 
the switch has no terminal with at least a one ir, its labe ^P«« 0 compensated predicted and 

SSS ;Ue S - P^" — 113 and 114 accordingly to the 

subtracter 112. Motion estimator £££^ZS?~ outputs a motion vector, i.e., a 

40 (F1) 125 and a nul. block from null block 127 ^to denve an ppy r ^ g ^ size b|ock , n 

displacement relative to the location * >^ ** each of the b,0CkS *» su PP ,ied . t0 

reference frame l m that best matches the boc^The motion n» ^ fof encodjng pnQf 

motion compensator 128 via ^f^^^^^TS 117. Motion compensator 128 aiso 
to being incorporated into BIT STREAM by multiplex and butter u generates a predicted 

< 5 receives the prior reconstructed frame ^urrentiy ^^^^J elemen t 104 to subtracter 112. 
version of the block in the S £*^ block from the actual current block and 

Subtracter 112 subtracts the predicted verier ' <^^^° 114 to dot 119 . dot 1l9t«»farm8tto 
generates error signa. E which is supplied ^j^* domain and the resulting transformed error signal is 
error signa. into the Discrete Cos-ne Transform errQ 9 r signal supplied as an output 

50 supplied to and quantized by ^ant.zer 120. ™£1SZ^ coder (VLC) 121 for encoding. The encoded 
SSM— S -Plied as an output to multiplex and buffer un,t 

117 Th e quantized transformed error signal ^^^^^^^ SS£ 
55 an input to dequantizer 123 through ^^J^^^J coefficients which are supplied 
on the quantized transformed ^^[T^Ua^i a reconstructed version of the error signa. ,n 

£7^^ * - - *- is added by adder 126 10 
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the motion predicted version of the block in the current frame received via switch element 113 to form the 
corrected predicted current block which very closely corresponds to the current block. This corrected 
predicted current block is supplied via switch 106 to frame buffer (F2) 130 and stored therein. The blocks 
which correspond to the current frame are accumulated in frame buffer 130 and become a reference frame 
5 in the future (l p ) for frames to be interpolated. The variable length encoded motion vectors and the variable 
length encoded quantized transformed error signals are then multiplexed together into signal BITSTREAM 
by multiplex and buffer unit 117 which may be transmitted or stored depending upon the application. Signal 
BITSTREAM can be employed by a corresponding decoder to reconstruct a representation of the original 
frame. 

io Subsequent frames, the QxR blocks of which are supplied as signal BLOCKS by control and frame 

rearrangement unit 101, are to be motion interpolated until a frame that exists at a later time than the 
second frame which was motion compensated predicted is to be encoded. Under the control of control and 
frame rearrangement unit 101 over first control bus 102, each of switch elements 102 through 110 has their 
arm positioned to connect to the terminal having at least a two (2) in its associated label or if the switch has 

75 no terminal with a two in its label the position of the switch is irrelevant. For clarity of exposition it is 
assumed that all blocks in the subsequent frames are to be motion interpolated and that block type unit 111 
"configures the arm positions of switch elements 113 and 114 accordingly to the terminal having at least a 
two (2) in its associated label. 

Each QxR block of pels of signal BLOCKS is supplied as input to both motion estimator 118 and 

20 subtracter 112. Motion estimator 118 receives a reconstructed version of the first (past) frame from frame 
buffer (F1) 125 and a reconstructed version of the second (future) frame from frame buffer (F2) 130. In 
accordance with an aspect of the invention, motion estimator 118 derives and supplies as an output a set of 
motion vectors, i.e., a set of displacements d m , d p within a predetermined area of the location of the block, 
wherein d m points to a same size block in the reference frame in the past l m and d p points to a same size 

25 block in the reference frame in the future l p such that a weighted sum of the values of the pels of the blocks 
pointed to by the displacements best approximates the block being encoded. The motion vectors are 
supplied to motion interpolator 129 via switch element 108 and to variable length coder (VLC) 116 for 
encoding prior to being incorporated into signal BITSTREAM by multiplex and buffer unit 117. Motion 
estimator 118 also supplies as an output to motion interpolator 129, in accordance with an aspect of the 

30 invention, a set of weighting values (weights) a m , a p for the values of the pels of the blocks pointed to by 
each of the motion vectors for the block, one weight corresponding with each of the motion vectors. Motion 1 
interpolator 129 receives the prior reconstructed frame currently stored in frame buffer (F1) 125, the 
reconstructed version of the second (future) frame from frame buffer (F2) 1 30, the motion vectors via switch 
element 108 and the weights via switch element 109 and generates a motion interpolated version of the 

35 block in the current frame. This motion interpolated version of the block is supplied through switch element 
104 to subtracter 112. The interpolated version of the block in the current frame is derived by multiplying 
(weighting) the value of the pels of the block pointed to by the motion vector for the first (past) frame by the 
weight for the past frame and adding that to the value of the pels of the block pointed to by the motion 
vector for the second (future) frame multiplied by the value of the weight for the future frame. Subtracter 

40 112 subtracts the interpolated version of the current block from the actual current block and generates error 
signal E which is supplied through switch element 114 to DCT 119. DCT 119 transforms the error signal 
into the Discrete Cosine Transform domain and the resulting transformed error signal is supplied to and 
quantized by quantizer 120. The quantized transformed error signal supplied as an output from quantizer 
120 is supplied as an input to variable length coder (VLC) 121 for encoding. The encoded representation of 

45 the quantized transformed error signal is supplied as an output to multiplex and buffer unit 117. The weights 
from motion estimator 118 are supplied via switch element 109 to optional quantizer 130 which quantizes 
the values of the weights and supplies the quantized weight values to variable length coder 116. Multiplex 
and buffer unit 117 supplies the encoded representation of the quantized transformed error signal in 
multiplexed fashion with the outputs of variable length coders 116 and 122 as signal BITSTREAM which 

so may be transmitted or stored, depending on the application. 

When a block belongs to a frame that control and frame rearrangement unit 101 has determined, based 
on the time position of the frame, is to be motion compensated predicted or motion interpolated, block type 
unit 111 may determine that it would require less bits to directly intraframe encode the particular block. This 
determination is based on a comparison of the complexity of the error signal and the actual block itself, 

55 both of which are supplied as inputs to block type unit 111. If block type unit 111 determines that it requires 
less bits to directly intraframe encode the particular block, switch elements 113 and 114 are instructed, via 
commands sent over second control bus 115, to connect to the terminal having a zero label. In addition, 
variable length coder 116 is responsive to commands from block type unit 111 to suppress the supplying of 

5 
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intraframe encoded, as described above^ Block type i unrt i m m gnd buffer 

selected. This indication is itself encoded by vanab k length ~deM 22 and supp ^ ^ 

m ™ jrsvs ir rjr^ sss; ?» — - » — — - 

because of the position of switch element 106. determines tnat a „ the frames between the first (past) 
When control and frame rearrangement unit 101 ^. determines hat an t ^ ^ 

and second (future) frames have been r^^J'SS „to frame buffer 125 to become the new 
in frame buffer 130, which is the future .mage frame -*™^XZT^ the old future frame, will be 
past image frame. Thereafter, a new future fra me further -n ^ fU ^ e| ^J new reference frames could 
motion compensated predicted and stored ^^fZ^^^.oM be intraframe encoded 
be selected by control and frame rearrangement unrt J01 th HWo ^ ^ ^ 

s^e — ' " ~ " 

^^-rre^rs 

image lattice of each of frames ^ and I from signa V D.N ^ ^ past felatjve to f , 
and second frames processed by v.deo coder 1 ^'n,. ^ Qf 2fJ3 a QxR 

which is to be encoded by motion compensated V Te *^™*\J'™™™ m ed A typical value for each of Q 
subset of set of pels 201. is the block for which a search ,s to be VP h a 

and R is 16. However, for ^ity and brevity of exp = Q r,d R are eac^ ^ ^ ^ 
value of 4. A search range, denoted by D max . s ™. m ^ ,mu ™ ,na rfisolacer ne nt that is a candidate to be 
units of pels), in either the horizontal or vertK, -*J*J ^SXX*£- via the sum of the 
motion vector. The total search area def.ned ^J^J^"^ whic h indi cates the potential area 
gray shaded region 204 and block 203. The ^^^^^ locat ions in previous frame 
fn which to find each pel of a candidate block : to frame 202 which will be searched for 

202 marked by an X defines the set of centers * ^^^'J^J 203 to a location marked by an 
the best possible match to block 203 ^^^T.SSS to be a motion vector for block 203. For 

frames L. I P and l„ of signal VIDIN (FIG. 1). Frame „ i« « reference frame in the future 

encoder 10Q P L, is a reference frame in the past relative ^ frame £ and P • an* rf r ^ for 

relative to frame .„. Block of pels 302. a QxR sub «*o ^ ^p,,,^. The reference 
block 302 that a search for motion vectors and ^respond ng weights s P tQ f ^ ^ 

frames are typically arranged such that one is in the past (U and ^^^M. *, of pels 201 , 202 
ITiThI^^ — 0. - « - - in the past 



45 



minimized. Such a minimum is expressed by 



50 min £ E 2 (x') (2) 

a m .a p .d m .d PX ' €N(x) 



55 where 

E(x') = l„(x')-(a m l m (x' + d m ) + a p l p (x' + d p » (3) 
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is the error for d m eD m and d p eD p . Respectively, d m and d p are the displacements for frame l m and l p . A 
candidate pair of values for d m and d p are graphically depicted in FIG. 3 as vectors 303 and 304, 
respectively. Also, D m and D p correspond to the search ranges, respectively, in frames frame l m and l p . In 
FIG. 3, the respective search areas defined by D m and Dp are graphically illustrated by shaded regions 305 
and 306. a m and a p are the weighting factors accorded to each pel value of the respective blocks to be 
combined from frames 201 and 202. 

In accordance with an aspect of the invention, the determination of optimum displacements and weights 
is achieved by imposing that the partial derivatives of 



10 



X E 2 (x') 
x'eN(x) 



75 



20 



with respect to a m and <* p equal zero. This leads to the following system of 2 linear equations 
M.u = v (4) 

where vectors u and v are given by 



[°J 



(5) 



25 



30 



V = 



and where matrix M is given by 



£ I m (x' + d m )I n (x') 
x'e N(x) 

£ I n (x')I p (x' + d p ) 

i'eN(i) 



(6) 



35 
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M = 



£ l2(x' + d m ) £ I m (x' + d m )I p (x' 

x'eN(x) x'eN(x) 

£ I m (x' + d m ) I p (x' + dp) £ i2( x ' + d p ) 

>'eN(») x'€N(x) 



+ d p ) 



(7) 
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Matrix M can be shown to be definite and positive for all values of x, d m and d p thereby guaranteeing that 
the corresponding optimal values of the weights, 



= a n 



50 
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as evaluated for any pair d m ,d p will lead to a minimum of the E energy for that particular d m ,d p pair. 
Singular values of M result in an infinite set of values o* and a* that satisfy equation (2). Such degenerate 
cases occur, for example, when there exists a perfect match for the block (no error) in both l m (x) and l p (x). 
For such degenerate cases any arbitrary value may be chosen as the optimum value for one of the weights 
and the optimum value for the remaining weight is determined by solving either of the two linear equations 
represented by equation (4). In non-degenerate cases, the optimal solution for the set of weights iT is given 
by 



(8) 
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selection of an pair d m ,d p that satisfies equation (4) and has fh« Howest E JJJf^ 
of candidate J 1, disp.acement pair d m .d p are then 

One such alternate measure is an absolute value type ,of f ^' on - |nven , on> several 

acceptably small value tor tne energy u« u » , threshold finds a displacement pair 

locations in each of frames l m and l p that are suDsiaruidi.y «h selected as a set of initial 

by approximately one quarter of the d,s* >nce o *e ^ ™£JZ^Z»cm* K 
45 candidate displacements. For example mead h of* 2°i * are djstant the 

A.so select the 8 halt the ^JSS wL mtimum disp.acement of D m and D, 

largest mteger value mailer than = one half me co p ^ ^ jn 

respective^ e.g 3 to^-Dp * Ej* . m ^ combjnatjon of all tne select e d 

of the set of pairs. The minimum c energy is co. *« m selected displacement pair, 

displacement pair with the minimum > E energy ^-J^nt defined by a .ocation in 

" b ° th N e n xt X £ ^"of SET SSSSr S ^ Tail removed. The displacements 

defined by^aTon: and d P - and those displacements defined by predetermined .ocat.ons ,n 
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each of frames l m and l p that are substantially equidistant from each location of set of locations 405 by 
approximately one eighth of the distance of the total search area are selected as a new set of candidate 
displacements. For example, in each of sets of pels 201 and 202 select dj? and and the 8 
displacements defined by locations around locations 405 that are distant by the largest integer value 

5 smaller than one quarter the component wise maximum displacement of D m and D p , respectively, e.g., 2 for 
D m = Dp = 7. The total set of locations 406 selected are marked in FIG. 4 by at least a filled square. 
Thereafter, a second set of pairs defined by the pairwise combination of all such selected displacements 
having one displacement defined by a location in frame l m and one displacement defined by a location in 
frame l p is determined. For this example, again, 81 candidate pairs of displacements will be members of 

io this second set of pairs. The minimum E energy is computed for each candidate pair of the second set of 
pairs. The pair with the minimum E energy is designated dj^.dp 0 . This selected displacement pair, has one 
displacement defined by a location in frame l m and a second displacement defined by a location in frame l p . 
The defining point of each displacement of the selected displacement pair with minimum E energy in this 
example is set of locations 407, location of which is marked by both a filled square and a circle in each of 

75 set of pels 201 and 202. 

Finally, the set of candidate displacements is again cleared. The displacements defined by locations 
407, i.e., dj? and d™, and those displacements defined by predetermined locations in each of frames l m 
and l p that are substantially equidistant from each location of set of locations 407 by approximately one 
sixteenth of the distance of the total search area are selected as the final set of candidate displacements. 

20 For example, in each of sets of pels 201 and 202 select dj? and d™ and the 8 displacements defined by 
locations around locations 407 that are distant by the largest integer value smaller than one eighth the 
component wise maximum displacement 6® and d™ respectively, e.g., 1 for D m = D p = 7. Each location 
of the total set of locations 408 selected is marked in FIG. 4 by at least a circle. Thereafter, a third set of 
pairs defined by the pairwise combination of all such selected displacements having one displacement 

25 defined by a location in frame l m and one displacement defined by a location in frame l p is determined. For 
this example, again, 81 candidate pairs of displacements will be members of this third set of pairs. The 
minimum E energy is computed for each candidate pair of the third set of pairs. The pair with the minimum 
E energy is designated d ®,d p 0 . This selected displacement pair, has one displacement defined by a 
location in frame l m and a second displacement defined by a location in frame l p . The defining points 409 in 

30 this example, of each displacement of the selected displacement pair with minimum E energy is marked by 
both a filled circle in each of set of pels 201 and 202. This selected displacement pair, having one 
displacement defined by a location in frame l m and a second displacement defined by a location in frame l p 
is selected as the motion vectors for block 402. In addition, in accordance with an aspect of the invention, it 
may be determined that this procedure for searching for a displacement pair to use as a motion vector 

35 should terminate whenever the E energy becomes equal to or less than a threshold value T. The 
displacement pair which first to achieves an E energy less than T is chosen as the motion vectors. Of 
course additional search steps could be added, as will be understood by one skilled in the art, depending 
upon the block size chosen and the error margin acceptable to the implementor. 

Another exemplary method for reducing the number candidate displacement pairs that need be 

40 checked to find a pair of motion vectors and their corresponding weights employs information concerning 
spatially neighboring blocks and individual pels, in accordance with an aspect of the invention, for arranging 
the displacement pairs that are members of the set of displacement candidates. The arrangement strategy 
is based on the generally true proposition that all blocks in a particular area of a frame are likely to be part 
of a single object which is moving. Therefore, all the blocks of which the object is comprised are initially 

45 presumed moving in a substantially uniform manner from frame to frame. Shown in FIG. 5 is a portion of a 
frame l n to be encoded by video coder 100 (FIG. 1) as it is divided into blocks of pels by control and frame 
rearrangement unit 101. Blocks of pels 501 and 502 are located at x (0 d and x (10) in frame l n which are 
located, respectively, to the left and above block 503 located at x <00 ) and which is currently to be encoded. 
The optimal set of displacement pairs, i.e., the motion vectors, are denoted for blocks 501 and 502 by 

50 

A = «4 01) , 4 01 >) and B = (d<]°\ d< 10 >) , 

respectively. In accordance with the assumption that for a sequence of images of natural scenes the motion 
55 field is smooth, an initial displacement pair, either A or B, is chosen for block 503, denoted by 

C = (d<°°>, <°>) 



9 



BNSDOCID: <EP 0511778A2J_> 



EP 0 511 778 A2 



10 



75 



20 



25 



nTough additional pairs of candidate disp ^« A S ^ itrch range having one displacement ,n each 
Employed so that a.. a displacement pair having the 

Srri^i -Sot ff^^X^^, alternately switch, the frame 

Se? and vice-versa. Such a method ™ motion interred encode* 

to be fixed initially is the one temporally < clo ^ S, h ^/^J es1imat e. One of two scanning orders may be 

™" — as C « — - - 

d„.o = d<?» anddp.0 = dT- 

s _ ,„ fig. e „. — ~ ^ WtEtW-'T 

ease opposition, it is assumed that d p is .nrt.ally fixed A.I values o P ^ djsplacements 

fna par w'h d p , are scanned in a circular fa^or .around JjJ^X^ numbers associated with each 
m 'selected for the exemplary search, depicted I n FIG. 6. . s, nd,ca y counter . c|ockwise or cloc kw.se 
point corresponding to each possible ^ e of d The electa ^ mined threshold T, the 

direction is arbitrarily decided. As soon as the E = as the motion vectors for 

search process is stopped and the current <^^J^ yMwA an E energy of less than T being 
block 603. In the event that all values of d m have been s ^ frame ^ an(J ca „ ed d 

found the value of d m leading to the m.n.mum o ( the J*™™ frame , around dpl . This p.ng- 

Soanning is continued, in a similar circular ^.on fw d.spl ™™ > Xq decrease . As switchin g between 
po'ng P-dure is iterated ^^ZZS ^sses, thereby ensuring that »e search w„. 
frames l m and 'p occur5> » im 57 ... 

C %X*er speed up the ^^"X" .^a^d Z^Z^Zs^ 
dpJ or all values of d p that are combined m «^^*%£ l « searched. Index j indicates the number 
are within the possible search range ^^^ 'rSe subset is selected so as to compose those 
of times each of frames l m and l P has ^J'^f dis '" ment of the frame for which the displacement 
d sptecements that are close in ^ £55?5om* neighbors to d mj and d m0 itself when 

is not fixed. For example, if it is deeded to only *«J^f tne E energy over the 9 possible pairs is 
d pJ is fixed, then the value of d m ref similar to that described above, 
kept constant while proceeding on to to extended, in accordance with an aspect of the 

The well known concept of conjugate direction searcn candidate displacements by 

invention, as another method of ^JS^aa displacements are searched in the finding 

determining the order in which members of the set ot ca _ |acements that are confined within he 

of moTion vectors. For purposes of examp£ al I ££j£ d ^ Shown in F , G s. 7, 8, and 9, in flow 
search range are included in the set of can d.splac«me ™ J\ ction search . The rout , n e is 

chart form is an exemplary i^P'e^tation °f the «^J*«Q when a set of displacements to 
executed by motion interpolator 129 (FIG ,) and t s entered v« > P^ ^ mjnjmum °! n this 

be used as motion vectors is needed for a btock^ in p & ^ reference frame , in this 

Stialized to infinity. Next, in step 803 the ^p, *»nertn djsp|acement se ^ ct e d for C as 
example frame U. is defined to r,ave the sam • row ^ defined by a search 

defined above and whose column value s set to ^ fow component D and 

ranqe D ma x- For purposes of this example, D m Dp Similarly, in step 804, the 

3S« component D_ Additionally row «ag - J Jg, . « tto ^ ^ ^ 

SrST-M ^m^ for Cad ^ >- -~ 

Essrsr r-ss t^^^^--* ™ ed for purposes of the 
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search. Similarly, the upmost and downmost directions can be uniformly interchanged for purposes of the 
search. Additionally, row and column order can be uniformly interchanged for purposes of the search. The 
directions in this example were selected for convenience purposes only. In step 805 the energy of E is 
computed for the current pair of displacement values and is denoted S. Next, conditional branch point 806 

5 tests if S is less than S min . If the test result in step 806 is YES, control is passed to step 807 which sets the 
following values: change__min, a flag indicating that Sm ln has been changed is set to 1; d m ,min» a variable 
that contains the current estimate of the displacement in frame l m , is set to d m ; dp >min , a variable that 
contains the current estimate of the displacement in frame l p , is set to d p ; and S min is set to the current 
value of S. Control is then passed to conditional branch point 808. If the test result in step 806 is NO, 

10 control is passed directly to step 808. Conditional branch point 808 tests if px_flag is set to 1. If the test 
result in step 808 is NO, control is passed to conditional branch point 809 which tests if d P(X is equal to the 
downmost row value D max , x . If the test result in step 809 is NO, control is passed to step 810 which 
increments the row value of displacement d p . Control is then passed back to step 805 to determine and test 
the new values of S. If the test result in step 809 is YES, control is passed to conditional branch point 811 

75 which tests if flag change_min is set to one. If the test result in step 811 is YES, control is passed to step 
812 which sets the new d p estimate to the row displacement found to have the minimum S and the leftmost 
column displacement. Additionally, flag change_min is set to zero (0) and px_flag is set to 1. Then control 
is passed back to step 805. 

If the test result in step 808 is YES, control is passed to step 813 which tests if dp, y is equal to the 

20 rightmost column displacement. If the test result in step 813 is NO, control is passed to step 814 which 
increments the column value of displacement d p . Control is then passed back to step 805 to determine and 
test the new values of S. If the test result in step 813 is YES, control is passed to step 815 which tests if 
flag change_min is set to 1. If the test result in step 815 is YES, control is passed to step 816 which sets 
the new d p estimate to the column displacement found to have the minimum S and the topmost row 

25 displacement Additionally, flag change__min is set to zero (0) and px_flag is set to zero (0). Then control is 
passed back to step 805. 

If the test results in steps 811 or 815 were NO, control is then passed to conditional branch point 817. 
Conditional branch point 817 tests if mx_flag is set to 1. If the test result in step 817 is NO, control is 
passed to conditional branch point 818 which tests if d m . x is equal to the downmost row value D max . x . If the 

30 test result in step 818 is NO, control is passed to step 819 which increments the row value of displacement 
d m . Control is then passed back to step 805 to determine and test the new values of S. If the test result in 
step 81 8 is YES, control is passed to conditional branch point 820 which tests if flag change_min is set to 
one. If the test result in step 820 is YES, control is passed to step 821 which sets the new d m estimate to 
the row displacement found to have the minimum S and the leftmost column displacement. Additionally, 

35 flag change_min is set to zero (0) and mx__flag is set to 1 . Then control is passed back to step 805. 

If the test result in step 817 is YES, control is passed to step 822 which tests if d m , y is equal to the 
rightmost column displacement. If the test result in step 822 is NO, control is passed to step 823 which 
increments the column value of displacement d m . Control is then passed back to step 805 to determine and 
test the new values of S. If the test result in step 822 is YES, control is passed to step 824 which tests if 

40 flag change_min is set to 1. If the test result in step 824 is YES, control is passed to step 825 which sets 
the new d m estimate to the column displacement found to have the minimum S and the topmost row 
displacement. Additionally, flag change_min is set to zero (0) and mx_Jlag is set to zero (0). Then control 
is passed back to step 805. If the test results in steps 820 or 824 were NO the current values of 
displacements d m , min and d p , min are returned via step 826. Thereafter, the routine is exited via step 827. 

45 As an alternative, a limited set of displacements, for example 3 or 5, could be compared in each 
direction, rather than [2|D m | + 1p, [2|D P | + 1] 2 respectively for each frame. 

In practice, an object present in the interpolated frame l n always exists in at least one of reference 
frames l m or l p . Therefore, the distribution of (a m , a p ) is such that 

50 a m +a p = 1. (9) 

The optimal solution to equation (2), can be expressed by 
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as long as 
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£ I m (x'+ d m ) - I P (x'+ d p ) *0. 

*'€N(X) 

20 d m = (n-m)/(n-p)dp (11) 

Might. .„ that yi.ids M !*'^.^^ ™ Thi SS J. K «, be desirabte 

Once en * ^'J^^^t^ tor ft* »M> « * «J* 

to quantize the »e,ghts y *, and IftereBy »™t t „ quamze d to a 

2-: .-sr? ;™ai« z» »~. <. «. * ~» « « ■ - 

"tr^l oCe^e to me .e, 9 h, S w <he E enengy ino.ease 4 tha, .a *-i»ed * a 
all analysis of the expression of the E emagy is g,»en by 

i . m,,(i«.) 2 *nK 2 (A.„) 2 *2m,!(4a»l(i«p) ( ,2 > 

wh ere m„ and m 22 are the ^na.*^ 

M. It is important to notice that equate ^j? ™\ nd _ direc tly from a reconstruction of 

that a decoder can estimate the Urease due'to quantization A max is known, 

frames L and l p as long the ""j^^^^J^ a quantization step size for « m . « P that 
For non-degenerate cases ^1*0). ,t ^ poss,We to ch Aa p in equation (12). The 

maintains A below a threshold A ma x- ,n,s ,s at,,IBVOU ' ' 
SSg quantization step sizes must be maintained smaller than 
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Aa n = Vm„A max /|M| . (14) 



Where the constraint of equation (9) is imposed, the maximum step size for A« m is given Py 
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It has been observed experimentally that the constraint imposed by equation (9) on the weight values 
causes the E energy to be much less sensitive to the quantization of « m , a p . Without employing the 
constraint of equation (9), larger step sizes for Aa m and Ao p are required due to the large values of rm 1 or 
rri22. 

5 Shown in FIG. 10, in flow chart form, is a simplified composite, in accordance with an aspect of the 

invention, of the steps required of motion estimator 118 (FIG. 1) to generate motion vectors and weights for 
a block to be encoded. Accordingly the routine is entered in step 1001 when it is determined that motion 
vectors and weights are to be generated. Thereafter, in step 1003 the current value stored for the error 
energy E store d, is initialized to infinity. In step 1005 candidate displacements are determined for each 

w reference frame. Step 1007 groups the candidate displacements into a set of groups d m ,d p . The next d m ,d p 
group to be processed is obtained from the set of groups. In step 101 1 M, M~ 1 and a vector v are evaluated 
for the d m ,d p group. Step 1013 derives iT = M~ 1 .v in accordance with an aspect of the invention. The error 
energy for ,« P ,d m ,d p is computed in step 1015. Thereafter conditional branch point 1017 tests to 
determine if the E energy is less than the current value of Es, 0 red- K the test result in step 1017 is YES 

75 control is passed to step 1019 which stores « p ,d m ,d p and sets E stored equal to the current E energy. 
Control is then passed to step 1021. If the test result in step 1017 is NO, control is passed directly to step 
1021. Conditional branch point 1021 tests to determine if E store d is less than threshold T. If the test result in 
step 1021 is NO control is passed to step 1023 which readjusts the set of d m ,d p groups. This readjustment 
can be a rearrangement of the order in which the groups are selected in step 1009, an addition or a 

20 deletion of groups from the set, whichever is required for an efficient search. Control is then passed to step 
1027. If the test result in step 1021 is YES control is passed to step 1025 which removes all remaining 
d m ,d p groups from the set of groups, thereby leaving no remaining groups to be processed. Control is then 
passed to step 1027. Conditional branch point 1027 tests to determine if there are any more d m ,d p groups 
remaining in the set of groups which need to be processed. If the test result in step 1027 is YES, control is 

25 passed back to step 1009 to get another group to be processed. If the test result in step 1027 is NO, 
control is passed to step 1029 for the performance of any optional quantization. Such quantization may 
actually be performed by quantizer 130. Control is then passed to step 1031 and the routine is exited. The 
final stored values of ,a p ,d m ,d p are available to be supplied as the motion vectors and corresponding 
optional weights. It will be recognized by persons skilled in the art that the above described invention may 

30 be employed on each component of a composite signal. 

Claims 

1. A method for developing an encoded signal of a block of pels to be encoded located at a 
35 predetermined location in a frame of an applied video signal comprised of frames, wherein each frame 

contains at least one image representation, by developing a set of signals representing displacements 
and corresponding weights from at least two frames of said video signal that are designated as 
reference frames and are divided into blocks of pels, comprising the steps of: 
a. storing said reference frames in a memory; 
40 b. selecting, in accordance with a predetermined selection criterion, for each of said reference 

frames, one or more candidate displacements, measured from said location of said block to be 
encoded, whose vector magnitude is less than or equal to the vector magnitude of a predetermined 
maximum displacement; 

c. culling, from said candidate displacements that have been determined, a set of groups of 
45 candidate displacements such that each group of said set has one candidate displacement member 

from each of said reference frames; 

d. ordering said groups of said set of groups in accordance with a predetermined ordering criterion; 

e. assigning to each group a set of best candidate weights for said group, one candidate weight 
corresponding to each member of said group; 

50 f. generating for each of said groups a composite error signal indicating differences between values 

of pels of said block being encoded and a weighted sum of values of pels of said blocks of said 
reference frames that are displaced, from said location of said block being encoded, by correspond- 
ing ones of said candidate displacements in said groups of said candidate displacements, said 
weighted sum being determined by multiplying corresponding ones of said values of pels of said 

55 blocks of said stored reference frames by corresponding ones of said set of best candidate weights 

for said group; 

g. storing said best candidate weights, said corresponding group and said composite error signal for 
a group of said set of groups having a minimum composite error signal, if said composite error 

13 
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signal is .ess than any previous* stored composite error signals or no composite error signal has 
been previously stored; pre determined condition is met by a group of said 

i. forming said encoded signal corresponding to sa d block from ^ 

for which said predetermined condition is met; and 

j. delivering said encoded signal to a transmission medium. 

The method as defined in Cairn 1 wherein ^^^ZteZ^^^ 
frames and for all iterations of sa,d step ^^^o 'Ta^eZ having different second endpoints. 
first endpoint at said location of sa,d block to be encoded, and ? as said location of 

said second endpoints being defined upon a firs « ^ £ ° distance from a 
said block and locations that are substantially ^XTfrom each other, and upon any 

predetermined set of distances ^ t ^^^^^J^ b y a second endpoint of a 

The . - deeded In Cto fiSZ^ZXZZ^ZZXZ 

set of distances. 

least one block adjacent to said block to be encoded. 

T h e method as define. In c,*n 4 — ^ ."p^— "~ 

of said step of selecting, aateang tot a pm* '«*' e ™»/|™ a ,J hion , I0m saia re ,a™oc. 
dlsplaoetnente. said P^J»' JtTSch ^noe traote no, a»d P^euta, 

recently had a minimum composite error signal. 

The method as defined in Cairn 4 wherein f ™~ ^f^ST^Z 
of said step of selecting selecting , f o ' *», said reference 

displacements, said particular reference J ^SS^^SS in an immediately preceding iteration of 
frames upon each iteration of said step of fleeing to jousl stored error signa ,s was not 

^JEFJSl rCrnSTame^rS pScS reference frame emp.oying a corresponding 
displacement of said stored group. 

. ™ « W d as de„t»d ,n Cain, 7 — ^^S^SZSJSSZ 
displacements trom said ^^'2^^^^^ <«~ "» ■ »»* <" ca " dMMe 
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elements arranged in columns and rows, one column and one row corresponding to each of said 
reference frames and arranged in a like order for both columns and rows, each of said columns and 
said rows having a label that identifies a block in said reference frame that corresponds to said column 
or said row and is displaced, from said location of said block being encoded, by a corresponding one 

5 of said candidate displacements in said group of said candidate displacements, wherein each of said 

elements are a sum of products of values of each corresponding pel location of blocks identified by 
said row label and said column label of said elements; 

evaluating each element of a vector v having elements arranged in a column, each row of said 
column corresponding to one of said reference frames and ordered in like manner as said rows of said 

to matrix M, each of said rows having a label that identifies a block in said reference frame that 

corresponds to said row and is displaced, from said location of said block being encoded, by a 
corresponding one of said candidate displacements in each of said groups of said candidate displace- 
ments wherein said elements are a sum of products of values of each corresponding pel location in 
each of said blocks identified by said row label and said block being encoded; 

jo deriving for each of said groups a vector u* by solving the system of linear equations M.u = v, 

wherein elements of u* represent candidate weights, one candidate weight corresponding to each 
member of said group and ordered in like manner as said rows of said matrix; and 

employing said vector iT as said set of best weights assigned to each of said groups. 

?o 10. The method as defined in claim 9 further including the step of quantizing each weight of said vector u* 
corresponding to said group of candidate displacements for which said predetermined condition is met 
to a different vector u q . 

11. The method as defined in claim 10 in which two reference frames l m and l p are employed and said 
25 predetermined amount is A max , mn and 17122 are diagonal elements of matrix M, and mi2 is an anti- 
diagonal elements of matrix M, Aa mi Aa p are quantization step sizes employed for frames m and p, 
respectively, and are required to be maintained less than or equal to, respectively, 



' A 



and 

35 



VmnAnuj/IMI. 



40 

12. The method as defined in claim 10 wherein said step of deriving further requires that the sum of all 
members of said vector u* be one such that said step size Ao is required to be maintained less than or 
equal to 



45 VA mu /(m n +m 2 2-2m 12 ). 



13. The method as defined in claim 1 wherein said step of culling requires that all displacements in each of 
50 said groups are interrelated by a function of the relative times corresponding to said image representa- 
tions within said reference frames. 

14. The method as defined in claim 13 wherein said function of time is a ratio of the difference between 
said reference frames and said frame of said block being encoded. 

65 

15. Apparatus for developing an encoded signal corresponding to a block of pels to be encoded located at 
a predetermined location in a frame of a video signal that is supplied as an input, said video signal 
being comprised of frames wherein each frame contains at least one image representation by 
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maximum displacement^ 18]; H.cniaroments that have been determined, a set of groups of 

-"SStr eesigning fe e,» g™P » - - -** * - " ^ 

„.i 9 W responding to each ™nnbe, of sa|d»~rt12^ lndicaling di „ OTnees 

„,eans lor generating lor each of aard A^/^.d sum ol values of pels of said 

wn ,,e„ vafuea of pele of said block being 3„ „ ^ ,«* b.in 9 encoded, by 
blocks of aeid reference frames M ere ^d droops of said o.ndirfefe MamfA 

corresponding ones of said candidate c £ res ™di„g ones of aeid .aloes of pels of said 

srss red^~sr— ss. - - - - - -~ - 

-Id bes, — - r £ co^g gr^ - - — - 

ratrj s :cr^r«Te„ s^* », no 0™ — - 

been previously stored[1 1 8]; Hinntinn ODerat jons of said means for selecting, said means for 

means for sequencing and coordinating opento*** s means ^ ^.^ ^ g fg 

culling, said means for ^^^T^?^ <I said candidate dis P ,acementS and S3 ' d 
manner until a predetermined condition is met Dy a 9*°«^ 

x ^ponding se. of bee, SdCS'fe'L, bkaek. by including in said encoded 

5^3 " £ptZ™£« eel o, bee, cdidafe weights fo, s*d group fd, 
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© The invention relates to developing a signal to 
represent each block of an image contained in a 
frame being interpolated in a motion interpolated 
system such that the error between the frame being 
interpolated and the reference frames is minimized 
without placing any predetermined limitation on the 
values of either the weights or the displacements 
employed as motion vectors. The weights and dis- 
placements required to achieve the minimization of 
the interpolation error are jointly determined on a 
block by block basis. A set of groups having a 
candidate displacement from each reference frame 
and, in accordance with an aspect of the invention, 
their corresponding best weights are determined. An 
error signal which would result if each member of 
the set of groups of candidate displacements from 
each reference frame and their corresponding best 
weights were employed to represent the block is 
evaluated. The group having the lowest error signal 
is either selected to represent the block or as a 
starting point for the selection of a new set of 
groups. By employing other aspects of the invention, 
the number of groups which need be evaluated to 



represent the block can be reduced from a predeter- 
mined maximum number. 



Rank Xerox (UK) Business Services 

J3-09/3-09/3.3.4) 



BNSDOCID: <EP 051 177BA3_L> 



BEST AVAILABLE GOP* 



European Patent 
Office 



EUROPEAN SEARCH REPORT 



DOCUMENTS CONSIDERED TO BE RELEVANT 

~[ citation of document with indication, where appropriate, 
j C etejo ry of relevant passages 



IEEE TRANSACTIONS ON COMMUNICATIONS FEB. 
1990, USA, VOL. 38, NR. 2, PAGE(S) 215 - 
222, ISSN 0090-6778 . 
Cafforio C et al 'Motion compensated image 
interpolation' 

* page 217, left column, line 30 - right 
column, line 45 * 

CONFERENCE RECORD EIGHTEENTH ASIL0MAR 
CONFERENCE ON CIRCUITS SYSTEMS AND 
COMPUTERS (CAT. NO. 85CH2200-4), PACIFIC 
rom/F CA USA 5-7 NOV. 1984, ISBN 
KSnSfeShSaS. WASHINGTON DC USA 
IEEE COMPUT. S0C. PRESS, USA, PAGE(S) 391 

Kappagantula S et al 'Motion compensated 
hybrid image coding' 

* page 393, left column, line 1 - right 
column, line 17; figure 4 * 

US-A-4 958 226 (HASKELL BARIN G ET AL) 18 
September 1990 

* column 2, line 18 - line 37 * 

PROCEEDINGS. 10TH INTERNATIONAL CONFERENCE 
ON PATTERN RECOGNITION (CAT. 
NO 90CH2898-5), ATLANTIC CITY, NJ, USA, 
ll-Tl JUNE 1990, ISBN 0-8186-2062-5, 1990 
LOS ALAMITOS, CA, USA, IEEE COMPUT SOC. 
PRESS, USA, PAGE(S) 140 - 146 VOL. 2 
Bergeron C et al 'Parametric block 
estimation of motion and application to 
temporal interpolation of video sequences 

* page 143, left column, line 39 - right 
column, line 32 * 



1-15 



The present search report has been drawn up for all claims 



Place of w*ca 

THE HAGUE 



D*e af coa*l«lfcM of Ike 

25 July 1994 



Relevant 



Application Number 

EP 92 30 3631 



CLASSIFICATION OF THE 
APPLICATION <lot.CLS) 



G06F15/70 
G06F15/64 



1-15 



1-15 



1-15 



TECHNICAL FIELDS 
SEARCHED (lnt-CLS) 



G06F 
H04N 



Pierfederici , A 



CATEGORY OF CITED DOCUMENTS 

X • particularly relevant if taken alone 

Y : particularly relevant if combined with another 

document of the sane category 
A : technological background 
O : noo- written disclosure 
p : intermediate document 



T : theory or principle underlying the invention 
E : earlier patent document, but published on, or 

after the filing date 
D : document dted in the application 
L : document cited for other reasons 



& 7 member of the same patent family, corresponding 



BNSDOCID: <EP_ 



_0511778A3_I_> 



